1. Field of the Invention
The present invention relates generally to an image analyzing apparatus and method for automatically retrieving image features, as well as a computer readable program record medium in which an image feature extraction program is stored, and, more particularly, to an image analyzing apparatus and method for extracting image layout, shape and texture features through the utilization of coefficients of frequency bands for wavelet transformation, as well as a computer readable program record medium having therein stored an image feature extraction program.
2. Description of the Related Arts
Nowadays, due to the broad use of digital images attendant on the spread of Internet, a lot of attention is paid to the similar image retrieval based on the image contents in a personal digital library, an Internet provider""s retrieval service, a public digital library, etc. For this similar image retrieval, it is conceived to extract features from an image. A lot of researches have addressed the image features, with the attention to the color, texture, shape, etc. For the color feature by way of example, a histogram in accordance with Reference 1 which follows has been used as the simplest feature.
[Reference 1] M. Ortega et al. IEEE Trans. Knowledge and Data Engineering, vol. 10, no. 6, pp. 905-025, 1998.
However, the feature based on the color histogram may lack intra-image spatial information. This often leads to a use of the image segmented layout feature therewith. There exist innumerable researches on the texture, some of which are based on the autoregressive model or frequency analysis. A recent report says that use of the wavelet transformation may result in excellent performance. An edge detection is basically and widely used for the shape feature. The edge detection is too sensitive to a displacement, which needs some modification.
However, in such a conventional feature extraction for image retrieval, etc., use has been made of algorithms dedicated to items such as layout, shape and texture, to extract the features. For this reason, it is necessary to perform the feature extraction from the key image for each of the items such as the layout, shape, texture, etc., thus making the image retrieval time-consuming.
On the other hand, a rapid image retrieval using the wavelet transformation is described in References 2 and 3 which follow.
[Reference 2] C. E. Jacobs et al. Computer Graphics Proceedings, Annual Conference Series, pp. 277-286, 1995.
[Reference 3] E. J. Stollnitz et al. xe2x80x9cWavelets for Computer Graphicsxe2x80x9d, Morgan Kaufmann Publishers, Inc. 1996.
However, these image retrievals using the wavelet transformation may suffer from a deficiency that the entire image is wavelet transformed to use the entirety of each frequency band as the feature, so that a retrieval is possible of only images totally analogous to the key image, thus making difficult a flexible retrieval, i.e., a retrieval of analogous images through the grasp of the individual features such as the layout, shape or texture possessed by the key image.
The present invention provides an image analyzing apparatus and method capable of a rapid image feature extraction and of a highly flexible feature extraction ensuring individual extractions of features such as layout, shape and texture, and provides a program record medium having an image feature extraction program stored therein.
The present invention further provides an image analyzing apparatus and method having a scale invariance capable of recognizing images having the same contents as the same one irrespective of the size and capable of retrieving in spite of interminglement of color images and gray scale images, and further provides a computer readable program record medium having an image feature extraction program stored therein.
According to a first aspect of the present invention, there is provided a image analyzing apparatus comprising a wavelet transformation unit for receiving images to effect a wavelet transformation thereon; and a feature extraction unit for extracting features of images separately in the form of layout, shape and texture features, on the basis of coefficients of frequency bands after wavelet transformation. For this reason, as compared with the prior art example in which the layout, shape and texture features have been extracted by independent algorithms, the present invention can achieve a high-speed extraction of features belonging to different categories by making use of the wavelet transformation to thereby unify the algorithms for extracting the layout, shape and texture features.
The wavelet transformation unit of the present invention performs a wavelet transformation for multiple resolution analysis after the determination of the decomposition number by the decomposition number determination unit so as to allow the size of the lowest frequency band of the wavelet transformed data to become smaller than a predetermined size. In this manner, the present invention utilizes the multiple resolution properties of the wavelet transformation, with the result that the features can be extracted without depending on the image size, and a scale invariance is presented that allows the images having identical contents to be recognized as similar ones irrespective of the size.
It is essential for the feature extraction unit to be able to extract features in the form of the gray scale images. To this end, it comprises a layout feature extraction unit for determining a layout feature from a coefficient of the lowest frequency band LL in the wavelet transformed data structure; a shape feature extraction unit for determining a shape feature from coefficients of the second lowest frequency bands HL0, LH0 and HH0 after wavelet transformation; and a texture feature extraction unit for determining a texture feature from the average and deviation of frequency bands HLn, LHn and HHn (where n=0, 1, typically n=0, 1, 2, . . . ) excepting the lowest frequency band LL after wavelet transformation.
In cases where the present invention deals color images, the wavelet transformation unit, upon reception of a color image in RGB color space by an RGB/YUV transformation unit, transforms the RGB color image received into a color image in YUV space to thereafter effect a wavelet transformation thereon. In case of the color images, the layout depends on the distribution of colors. The layout feature extraction unit thus has only to consider a deal of the color at the roughest resolution region in the wavelet transformation. The coefficient of the lowest frequency band LL after wavelet is transformed into a color representation in the other color space matching the human senses, to thereby determine the layout features. The shape and texture features involve no color concepts. Thus, in case of using YUV as the color space, the features of the luminance components Y on which the most amount of signals may concentrate are retrieved. That is, the shape feature extraction unit uses only luminance values Y of YUV components constituting the coefficients of the second lowest frequency bands HL0, LH0 and HH after wavelet transformation to thereby determine the shape feature. Furthermore, the texture feature extraction unit determines the texture feature from the average and deviation of only luminance values Y of YUV components constituting coefficients of frequency bands HLn, LHn and HHn (n=0, 1, 2, . . . ) after wavelet transformation. The layout feature extraction unit transforms the coefficient of the lowest frequency band after wavelet transformation into a color representation in HSV color space, Lab color space or Munsell color space to thereby determine the layout feature. The layout feature extraction unit normalizes the lowest frequency band after wavelet transformation into a predetermined size. This ensures a scale invariance with which the images having the same contents can be recognized as the same one in spite of the different sizes. More concretely, the layout feature extraction unit interpolates shorter sides of the lowest frequency band LL after wavelet transformation to thereby achieve a size normalization. With respect to three bands HL0, LH0 and HH0 in each of which is arranged the coefficient of the second lowest frequency band after wavelet transformation, the shape feature extraction unit finds an absolute value of each coefficient and averages coefficients at relatively same locations within the bands into one band coefficient. This allows the layout feature and the shape feature to result in the same amount of data. The texture feature extraction unit generates as texture features a vector whose vector components are an average and a deviation of coefficients of frequency bands of contour vertical components HL0 and HL1 excepting the lowest frequency band LL after wavelet transformation, a vector whose vector components are an average and a deviation of coefficients of frequency bands of contour horizontal components LH0 and LH1, and a vector whose vector components are an average and a deviation of coefficients of frequency bands of contour diagonal components HH0 and HH1.
The image analyzing apparatus of the present invention quantizes the values of the layout feature, shape feature and texture feature that have been determined from the coefficient of the frequency bands after wavelet transformation, into one byte length, to thereby reduce the amount of data for speedup. The image analyzing apparatus further comprises a similarity calculation unit figures out similarities between two images with respect to the layout feature, the shape feature and the texture feature that have been extracted by the feature extraction unit, the similarity calculation unit imparting weights to the similarities to figure out a total similarity between two images. By varying the weights to be imparted to the similarities of the layout, shape and texture, a highly flexible retrieval depending on the object can be achieved. For example, by rendering only the similarity of the shape feature effective, it is possible to retrieve images of, e.g., analogous but white-black reversal posters or of vehicles of the same type but different in color.
Alternatively, by rendering only the weight for the texture feature similarity effective, it is possible to retrieve images analogous in roughness of fabric texture or images of minerals or fossils having similar surfaces.
According to a second aspect of the present invention there is provided an image analyzing method comprising the steps of entering an image for a wavelet transformation; determining a layout feature from a coefficient of the lowest frequency band after wavelet transformation; determining a shape feature from a coefficient of the second lowest frequency band after wavelet transformation; and determining a texture feature from an average and a deviation of frequency bands excepting the lowest frequency band after wavelet transformation. The details of the image analyzing method are the same as those of the image analyzing apparatus.
According to a third aspect of the present invention there is provided a computer readable record medium having an image feature extraction program stored therein, the program comprising the steps of entering an image for a wavelet transformation; determining a layout feature from a coefficient of the lowest frequency band after wavelet transformation; determining a shape feature from a coefficient of the second lowest frequency band after wavelet transformation; and determining a texture feature from an average and a deviation of frequency bands excepting the lowest frequency band after wavelet transformation. The details of the computer readable program record medium having therein stored the image feature extraction program are also the same as those of the image analyzing apparatus.